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ABSTRACT 

fly and large behavioral scientists, particularly in 
education and psychology lack the sophisticated mathematical 
understanding necessary for adequate statistical analysis, especially 
multivariate analysis. And incompetent use of measurement theory, 
psychometry, and statistics cannot but lead to bad research. The 
remedy for this is better mathematical training of the behavioral 
scientist and the creation of special well educated advisors on 
research methodology. The advisor's role is not only one of dealing 
with attitudes to research methodology and the consequences of 
inadequate methods, but also to f artlier the methodological education 
of the client. Preventative measures must be taken to keep 
researchers in education and psychology from slowly dividing 
themselves into two non-communicating camps depending on the 
knowledge of mathematics and statistics. The problems are solvable 
only if more researchers are willing to change their attitudes to 
methodology. (MC) 
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This paper discusses the poor conditions of research methodology 
in the behavioural sciences due to insufficient consumer knowledge 
and lack of well-educated advisers in this area. It starts with a 
definition of research methodology, examplifies the kind of 
knowledge needed by the adviser and then goes on to the main 
subject, that is, different aspects of the adviser's tasks. Among 
other things, it deals with attitudes to research methodology, 
the consequences of inadequate methods, the methodological 
education of the client and the adviser's own research. Most examples 
are taken from multivariate analysis. 



Paper read at the conference of Society for Multivariate Experimental 
Psychology in Nijmegen in September 1972. 
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INTRODUCTION 

Research methodology can be defined very broadly, to include such 
things as the construction of proper concepts, theories and hypotheses 
and the usefulness of and similarities between different models. 
In this paper a more limited view is taken and research methodology 
refers to generic methods of collecting and processing data. They 
can usually be categorized as follows: 

1. Design 

2. Sampling technique 

3. Instrument for data collecting 

4. Statistical analysis 

5. Computer use 

The first point concerns the architecture of an examination, 
e.g. the possibilities of performing an experiment and advantages and 
drawbacks of different ce signs. V/e have partly the general type 
of discussion to be found in Bracht & Glass (1968) and Campbell & 
Stanley (1963), partly the more technical considerations as they 
appear in Cohen (1969) and Winer (1962). 

The second point seems to be of different importance to different 
researchers. I do not intend to argue on this issue, but the fact 
remains that not many educational reports can boast of random 
samples, not to mention simple random samples, which most 
statistical tests assume. 

The third point comprises measurement theory and psychometry, 
e.g. problems of construction and evaluation in connection witli 
questionnaires, tests, ratings and interviews. As far as application 
is concerned, this is an underdeveloped area and I have argued 
elsewhere (Larsson, 1971) that several differences of results may 
very well be the effect of bad measurement theory. However, I do 
not wish to discuss it here, but instead concentrate on point 4. 

This point is intimately connected with the first one and contains 
a large number of problems which are central to mathematical 
statistics. This paper will for the most part illustrate its discussions 
by examples from this area, particularly from multivariate analysis. 

Point 5 may concern the development of programs, problems of 
choice between different facilities of both sroftwarc and hardware, 
and the corr.puter use for simulations and documentation. 
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Rccearch methodology ic more or less based on mathematics. 
The degree of mathematical involvement is dependent on the particular 
field of interest. V/ell-defined area.n, e.g. most areas within physics, 
have usually no need for any fancy statistical methods to discover their 
experimental effects. V/hiie th^c need not necessarily be true, the 
opposite will certainly hold. One cannot but expect email effects in 
experiments which are founded on insufficient meauurornent theory 
and which are rather uncontrolled, as is often the case with research 
in education and psychology. No statistical method can eliminate these 
bad conditions, but while the behavioural scientist tries to make them 
better, we may hope, he is (or should be) a big consurr.cr of statistics, 
which helps him to discover the effects, if any. 

And this is where troubles start. Unless the behavioural scientist / 
is very specialized, he needs several kinds of statistical ammunition 
to hit his targets. This requires a rather profound knowledge of 
both applied and mathematical statistics, a knowledge which he seldom 
posesses. (I could add knowledge of measurement theory, of 
psychometry, and so on, but let us mainly limit ourselves to statistics.) 
The fact is also valid for most postgraduate students. Some readers 
may smile sceptically and say that his students have sufficient knowledge. 
Well, I do not know what super-students some readers may have, I can 
only say that I have not met them. In fact, I do not think it is realistic 
to let all students have this knowledge. After all, they intend to do 
behavioural research, not to become statisticians. 

It is perhaps convenient to say something about the kind of knowledge 
I am thinldng of before proceeding/ with the discussion. If the behavioural 
scientist wants to guide himself properly, as concerns the choice of 
statistical methods, he must know how and vhy they are constvuctcd, 
e.g. to understand their limitations. I doubt whether books like Cooley 
& Lohnes (1971), Hays (1963) and V/iner (1962) are sufficient for this 
purpose. Parts of the content of e.g. Dempster (1969), Ivliller (1966), 
Morrison (1967) and Rao (1965) must also be understandable. As examples 
of somewhat more specialized books on the same mathematical level, 
wc may mention Cox & I/iller (1965) and De Groot (1970). To this we 
may add books dealing with methods which by tradition belong to the 
behavioural sciences: Bock & Jones (1968), Horsf (1965), Lawley k 
Maxwell (1963), Lazarsfeld & Henry (1968), Lord & Njvick (1968) and 
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Rozcboom (1966). Good working knowledge of eJemcntary calculus 
and matrix algebra is usually inclispensiblc in this connection, Merc 
advanced Icnowledge can also be useful, v/hen rcadin^r e.g. Dempster 
(1969). 

V/hat is the actual knowledge of the postgraduate student and the 
researcher of the behavioural sciences? I have limited experience 
from Ccandinavian educational psychology only, but I do not think 
that things are much different elsewhere, Thoup,li there is a great 
variation among people, the l;nowledgc is, on the average, certainly 
not on the level exemplified above, j^/iost students and researchers 
have a dim apprehension about calculus and it is very fev/ who know 
anytliing essential about matri:: algebra. Unfortunately net too seldom, 
you meet behavioural scientists v/hcse knowledge of mathematics 
beyond the cquare root is almost infinitesimal. It is cbviouc that 
statistica.1 understanding will not devci>;jp easily under such conditions, 
and this is particularly valid for multivariate analysis, Co there is in 
general a wide gap between the knowledge A^hich the scientist really 
has and that which he should have, if he wants to be his own statistical 
adviser, 

ADVISING ON RESEARCH METHODOLOGY 

Since I have already said that it v/ould be unrealistic to assume the 
latter kind of knowledge in every researcher, the reader may have 
anticipated my view that we need special advisers on research 
methodology within the behavioural sciences. But this ic'ea will fail 
unless the adviser can communicate adequately with his client, 
and whether this is possible or not depends on several factors, I shall 
discuss some of them, mainly the kind of work vhich the adviser must 
do, the knowledge and attitudes of the client needed for efficient 
communication with the adviser, and some of the consequences of 
certain assumptions not being fulfilled. The discussion will start 
from the adviser's different tasks, and for each task I will incorporate 
my opinions about the client where convenient. 

If the cooperation between adviser and client is to give good 
rcL'ults, this implies considerable work by the adviser. He should 

1, propose and describe methods to the client, 

2. interpret output from computers^ 
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3. be co-author of ropcrtc contaminrr problems with ctrong methodo- 
logical anchoring, 

4. communicate with prograrnmerc and other adviaerc of the client, 

5. keep himself informed on literature of research methodology, 

6. take part in conferences and other contacts of irnpcr:ancc for 
research methodology, 

7. do his ov/n research. 

Some aspects on the choice of methocs 

The first task is, of course, important and perhaps the first one which 
comes to mind when cpeakina about an adviser for this area. The second 
task is quite similar to the first one so we can just as well discuss 
both pointo simultaneously, I have separated the two points, so that 
nobody v/ill believe that the client is away for good, once he has been 
sent on to the programmer. Even if we pretend that the adviser has 
all the knowledge, which is required for a good adviser, there are 
other tilings that must function. The adviser must have time and the 
right attitude, ^he client must have sufficient knowledge and be motiva- 
ted. Also, the client must know his ov/n problems so that he can specify 
what he wants from the adviser. Of course, this is reciprocal in that 
the adviser must also understand the client's problem. It must further 
be underlined that the client n-j;USt seek advice in time and not wait 
until data arc collected, which unfortunately many clients do. 

If the adviser has not time cuough to explain methods and/or if 
the client is not able to understand the 'explanation', two things can 
occur: either the adviser realizes the facts and proposes a simpler 
method, which the client can comprehend, or the client leaves the 
adviser, no wiser than when he cair?o (which may moan that the client 
uses the proposed method without understanding it). The latter situation 
can occur if the adviser makes wrong assumptions concerning his 
clients' knowledge, but it can also occur when the client visits the 
adviser without having any 'honest' intents. He may have a firm 
resistance to learning a new method and expects to find ready-made, 
easily understandable methods, v/hich solve all his problems without 
compromises or any substantial effort from him. In fact, certain 
clients may use an adviser only to verify their own prior opinions and 
work and do not listen to anytliing which will question these. Cuch atti- 



6 



- 6 - 



tudec do definitely not produce nrood rcccarch and perscnc having them 
are more or less impossible to advice. 

Forgetting such cases \vhif.h I hope to bo exceptional, one may 
aok 'V/hat are the consequences of proposing ir.cthods which tho client 
can easily understand or already knows?' It is extremely difficult to 
state anytliing generic about ccnspquences. Perhaps one dares to say 
that only advising the client about methods from his own repertoire 
- though creatively - will too often pioduce research of insufficient • 
quality. These advices typically keep the client away from applications 
of multivariate methods, as these arc the methods about which the 
behavioural scientists usually have little knowledge. Perhaps somebody 
has tlie objection theit this ic not true with factor analysis, which is 
widely known among researchers of the behavioural\sciences. Personally, 
I do not believe in such a statement. I agree that factor analysis is 
widely used, but this is not a criterion for an understanding of the 
method. In fact, the question of consequences is more complex than 
that above, because you can add 'V/hat are the consequences of propo- 
sing convenient methods which the client is not able to use properly?' 

Let us take an example. A client's problem is such tliat a good 

advice would be to let him perform a pxq factorial discriminant analysis, 

for which v/e suppore a convenient program is accessible. The client 

has good knowledge of elementary statistics up to 'the t-test level' and 

he knows something about ANOVA but nothing at all about discriminant 

analysis. Both the adviser and the client are motivated to do tiieir best, 

but 'the adviser has not much time to devote to each client, \7hat is the 

'best' advice in such a situation? For the sake of simplification^'let 

us assume that the choice is between four alternatives (v/c have r 

dependent variables): 

2 2 

1. Perform r (p + q - p - q)/2 t-tests. 

2. Perform r pxq factorial ANOVAc. 

3. Perform one pxqxr factorial ANOVA. 

4. Perform one pxq factorial discriminant analysis. 

The first alternative implies chat the client hiinself can handle the 
output, although it can be a tremendous lot of output. Besides the 
amount of output three other features are clearly negative: no interaction 
tests, the mass significance problem and the correlations between the 
dependent variables. As far as I know, it is very difficult to state how 
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they in detail contribute to wrong decisions, only that they contribute to 
wrong decicicns. The second alternative produces into i action touts, 
mitigates the mass significance probleir., but not the diird one. The 
third alternative is equivalent to the f-:urth, provided that the covariancc 
matricec for the dependent variables are all of a certain form (from 
cell to cell in the pxq factorial design). 

On the other hand, the client is uncertain about come pvcpertics 
about a p::q factorial ANOVA, e,g, how to perform contrast analyses, 
and ctill more uncertain about the more complex ANOVA v/ith repeated 
measures, not to mention discriminant analysis. Tliore are m.any 
technical details which must be considered for thia situation, but it is 
evident that the adviser must co:-r.promice, which mean:; that alternative 
2 or 3 ic probably chosen. To put it bluntly: what is the most serious 
error, to suggest a sub-optimal method which the client can use, or to 
suggest a good method which the client does not undci stand? Indeed, I . 
do not know; and there is probably no general answer to tliis question. 

The example above could be multiplied, but I shall only take one ^ 
further e::ample. A client has a problem which fits factor analysis of 
variables and he has a determined opinion about what kind of a structure 
he expects. The client has a very vague apprehension of factor analysis 
and the adviser has two different programs to offer, again for the sake 
of simplification: 

1. The principal axis method with varimax rotation, see e.g. Dixon 
(1970). 

2. A general program for analysis of covariance structures, as this 
theory is described in J6reskog (1970). 

Although tlie mathematical development of both rnetliods are 
difficult and because of that are beyond the usual client^'s capacity to 
understand, the second method is far more comple::. But the first 
mctliods is not very acceptable in this case, since the client has a 
special hypothesis about structure, and this cannot be tested by the first 
mctliod. If he uses the first method he could very v/ell cheat himself to 
believe that his hypothesis is correct, while the other method had given 
him a significant result. (The rotated factor matri:: seems to be 
'correct' , but the second method reveals that the hypothesized structure 
is insufficient to properly reproduce the covariance matrix.) Regarding 
that the client has, under all circumstances, bad knowledge of factor 
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analycic, I think that the cidvicov can hero juct as well choose the 
more appropriate methocU The only real alternative ic; to frankly 
assert tliat the client himself is not capable of coinr; any factor analysis 
at all, an assertion which perhaps will not be accepted. 

I assume that the reacljr finc'.s the sketched situation to be uucatis- 
fying. li CO, I am satisfied, because that was what I intended lo show. 
It is my opinion that lack of c::pertnec3 on research methodology in 
education and pcycholot^y makes much research unnecessarily bad. 
How many results are by this reason not discovered and how many facts 
are juct artefacts ? For instance, hov/ many differences of results in 
the same area mirror methodological differences only? V/ho has ever 
measured the consequences of a bad methodology and, even v/orse, how 
many decisions can have been made on false proundc, e.r. due to 
significances based on nerrli3ible effects? Guch decisions can have great 
economic consequences, when concerning many people. I am frightened 
when thinking of it, because my subjective probability of being wrong 
in this case is too far from x.O. I can only hope that son-.ebody will 
convince mc that there is nothing to worry abcut. 

Some aspects of the instruction on research methodnlof-y 

Meanwhile v/o can return to the car^iy troubles, where we left them 
before the last paragraph. One of tlic facts is that the fev/ available 
advisers on research methodology have too little time for each client 
and an easily suggested remedy is to recommend convenient literai-ure 
which describes his problems. It is a nice thought but it will not work: 
many aspects of his problems are so technical that it will be impossible 
to find literature which matches the client's knowlcd^re. For several 
problems I think that the literature can be produced, but it is also 
evident that sor problems arc so tcclinical and/or some clients' know- 
ledge is so deficient that it is almost impossible to write a book of 
normal size without assuming prerequisites of the reader. For instance^ 
I can mention three books on multivariate analysis which were published 
by educational psychologists in 1971: Coolcy & Lohnes (1971), Tatsuoka 
(1971) and Van do Geer (1971). Although these books vary concerning 
mathematics, none of them is accessiblb to most behavioural scientists, 
as far as I can see. Also, books dealing with measurement theory 
and psychomctry have with yearo become more mathematically involved. 
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Compare e.g. Torgerson (1958) with Bock & Jones (1968) and CuUiksen 
(1950) with Lord & Novick (1968). 

One part of the formal education of research methodology, at 
least on the postgraduate level, should be courses in mathematics for 
those who do not have the necessary prerequisites here. I am aware 
of some institutes which have taken the consequences of this and offei 
courses in mathematics up to the level of elementary matrix algebra. 
However, too many institutes have no such courses, and I cannot teU 
whether this depends on economics or other factors. V/ith regard to the 
very shifting mathematical background students in the behavioural 
sciences have, these courses should be compulsory for those who need 
them. Otherwise much teaching on research methodology will be 
meaningless, and this goes for le :tures as well as for individual advice. 
It is quite naive to think that anybody can learn anything, no matter how 
mathematically involved, without spending a good deal of time on first 
learning mathematics. It ought to be self-evident, but my experience of 
many people'^s reasoning says it is not so. 

Cfi the other hand I am no friend of too much formal teaching on 
rese-^.r. rr^ .T^ethodology for students, at least not in the almost ridiculous 
way It is sometimes performed in Cweden. Often the students, and 
particularly the postgraduate students, are fed with an overwhelming lot 
of methodological facts within a few semesters. Most of the facts meet 
the student before he has done any substantial research and are then 
supposed to last for the rest of his life. Then the only thing that really lasts 
for many students is a feeling of distaste for everything that smells 
statistics. I can understand them: it requires an interest in methodology 
well above average to survive in such a climate. 

The only methodological course which I think is necessary at the 
postgraduate level, except .a course in mathematics, is one which gives 
a broad overview and not details of different methods, etc. It should give 
syntheses, point to differences and similarities of methods, discuss 
strategies of choosing methods. The more profound knowledge should 
be left to the individual adviser to give to the student (researcher), when he 
is motivated by his own research work to go into details with ir.t thods 
suggested. Of course, this mea.i>s that the adviser must spend rather 
a lot of timo with each client, and this would be more expensive than the 
arrangement of formal courses. However, I am quite convinced that the 
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individual way produces better applications of research methodology and 
does it more rapidly. 

It is not at all surprising that the researcher now and tlicn has use 
for courses on special subjects, including methodology. Particularly if 
several members of one or more institutes are iatci octed in chc same 
subject, there are good reasons to start a course. I have a feeling that 
this type of courses is not very common. Maybe there is a lack of 
motivation to further educate oneself, because k \v researchers have 
permanent appointments? (At least so in Sweden, where most people 
doing research have one yeai appointments.) I think that formal courses 
are much more advantageous to the researcher than to the student, 
because the former's work experience makes it easier for him to assimila- 
te the new knowledge. I also think that such courses can be efficient if 
you want to introduce a new method, etc. at your institute, but should 
be followed up with individual advising. The adviser on research methodolo 
gy has an important role in presenting, statistical innovations to other 
members of his institute. 

The time lag between the development of statistical ideas and 
methods and its applications in education and psychology is often sur- 
prisingly great. A typical example is the concept of power. Although 
the idea of power and statistical power analysis is over foity years old, 
it seems to have been noticed by few behavioural researchers. Maybe 
tables such as those by Cohen (1969) will give rise to a more extensive 
use of this important concept. And even if an idea has been introduced 
to some behavioural scientists, it can take time before it is firmly 
settled. As a probably extreme case I can tell that about three years 
ago I began to give the advice that Hays' q can be a good complement 
to the usual AN OVA output. It still happens that people ask me what this 
index implies. I presume that if I had had a one hour^c lecture on this 
concept for the members of our institute this would have raised the 
efficiency of the information spread considerably. 

One reason for the time lag with the multivariate methods is that 
they had to wait for the computer revolution. One prominent instance 
of this is canonical correlation* analysis, which, in spite of its introduc- 
tion by Ho telling (1935) to educational psychologists, had very few applica- 
tions in tlie behavioural stietices beifore I960. The lack of computers, 
which prevented many reoeax-chcrs from using multivariate methods, 
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while they were in their moct formative years, hac produced a gcnoratiou 
which ctill does not uce thece methods very extencivoly, because they 
have got other methodological prejudices, which arc not easily moved. 
Another essential reason for time lag is the mathematical barrier. 
Multivariate methods are, alr/iosv. by definition, not easy to understand. 
There are so many relationships which are difficult to sec through, 
e.g. because we cannot visualize more than three dimensions and 
that even such a simple concept as variance has no unequivocal extension 
to more dimensions. It must also be admitted that sevcial theoretical 
problems still wait for their solutions. 

The theoretical problems of multivariate analysis arc of two kinds, 
which are interdependent, viz. inference problems and data-analytic 
problems. Mathematical statisticians involved in this area are mostly 
concerned with inference problems. The direction of this research has 
been criticized (see Aitkin, 1971, p. 233) for dealing almost :.mtirely 
with methods which discover effects and almost never with methods which 
examine the nature of the effects. Problems of data analysis are more 
or less incidentally worked upon and comprise the diffirnlties of presen- 
ting data to human receivers in intelligible ways. Cavagc (1970) believes 
that data analysis will one day have its own founda.tion. 

The problems of data analysis arc also the problems of the adviser, 
when he v/ants to explain methods or ideas to his clients. He must do 
this in a non- technical way, while nevertheless retainirj the vital points 
of that which he is explaining. How this should be dene - when it can 
be done - must be a judgment from situation to situation. However, some 
general statements may be made, of v/hich I want to mention two. Try 
to translate technical or otherwise new concepts into concepts which the 
client already understands. Try to emphasize similarities between 
methods rather than differences, at least in the first place. Both 
recommendations rest on the assumption that you should take advantage 
of tlie knowledge which the client already has achieved. 

Factor loadings can be regarded as correlations in i-nany applications 

and eigenvalues can be regarded as variances here. You can inform the 

client that canonical correlations and, therefore, multiple correlations 

2 

are special types of product-moment correlations. Likewise, co is an 
ordinary squared multiple correlation between the dependent variable 
and the dummy variables defining the effect. And for a 2 x C contingency 



12 



- 12 . 



tabic Crarncr'c ctatistic (sec H<-iys, 1963, p. 606) io identical to 
and thereby the multiple corrolaticn, while this ctatictic ic noi; a 
canonical correlation in the general cace. If convcnicni:, prefer 
Bonferroni t-tests to other tectc for contrast analyccs (cee o.g. Miller, 
1966, pp. 67-70). When it comec te rr.ultivariate teste r.bout means, 
RoyV, union-inter section principle ic illuminating (gcc k'orricon, 1967, 
pp. HO-ltv and several other pzii^c:;). It has the further advantage of 
giving a simple view oi; hov/ c.iccriminant analysis ic connccued with 
MAKOVA. E.g. if you want to explain the multivariate t-test for two 
independent camples, the explanation could run like this: 'If these 
dependent variables are weighted together, you get a new dependent variab- 
le on which you can perform a t-test. Let us assume that you choose the 
weights in such a way that no other dependent variable, produced by 
weights, has a greater t-tcct value. If, in spite of this choice, it does 
not give a significant t-test, then we can say that the two groups arc 
not different, simultaneously for all original dependent variables. ' On 
the other hand, multivariate concepts like the generalized variance are 
not easily explained. This variance is defined as the determinant of a 
covariance rx-iatrix and is the product of its eigenvalues. Neither the 
variance equivalence of the einrcnvalue nor the connection of the deter- 
minant to certain volume? make this concept more understandable to 
me. 

The similarities of methods are not so easily pointed out. That 
multiple linear regression analysis is a special case of canonical 
correlation analysis is perhc )S one of the easiest similarities to mention. 
Perhaps also that discriminant analysis is a sort of specialisation out of 
MANOVA. Gome what harder to chow for a client is the connection 
between multiple linear regreccion analysis anc. ANOVA and ANCOVA. 
Paperc like that of Cohen (1968) can be useful here: tlie icloa of dummy 
variable coding is essential. If v/e e::tend it to multivariate analysis this 
means that canonical correlation analysis is the core of many methods. 
For latent structure analysis, the axiom of local independence is a common 
property to all its methods and rnay be used to emphasize similarities. 
It is also important to emphasize similarities and differences between 
different designs, e.g. exp^fiining advantages and drawbacks of the 
following four alternatives,, where experiment group/control group 
always is one factor and you have measurements both before (x) and after 
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(y) the c::porimcnt: 1) x/y is a second factor with repeated measures; 
2) X is categorized and becomes a second factor; 3) only one factor with 
dependent variable y-x; 4) only one factor with dependent variable y 
and covariate >:. I am strongly convinced that very much remains to be 
done on similarities and differences between methods. Thic involves 
m.athematical analyses as well as simulations and e::periencc'G from 
real data. I also believe in a better 'marketing' of such studies, so that 
they can be at least partly accessible to other than experts on a certain 
subject. 

The adviser's third, fourth, fifth and sixth task 

The third point of the adviser's working list (p. 5 above) is iraportaut, 
for two reasons. The first one is that it gives the adviser some possibili- 
ties of incorporating methodological studies here and there v^ithin the 
client's projects. While this in not often convenient, the opportunities 
should certainly not be neglected. The second reason is that some 
research reports contain rather advanced methods, v/hich can be too 
difficult for the client to describe and use properly alone. This is in line 
with what has been said before and to be able to compromise between 
good methodological advice and faulty consumer knowlerlgc, the adviser 
should write certain parts of the reports when necessary. It is then 
important that the adviser know his readers. Although a complex matter 
cannot be simplified too far v/ithout losing its essential properties, 
much more than what is done today can be done to facilitate understanding. 
To hide oneself behind a mathem.atical iron curtain, v/hen writing for a 
broad audience, is either a sign of ignorance or boastfulness. Of course, 
this takes time and implies that one adviser cannot serve many clients, 
but on the other hand it will raise the methodological quality of a number 
of reports. 

In connection with the fourth point, I would like to stress the important 
role of the programmer. Every sufficiently large research institate ought 
to have its own programmer. It is my definite opinion that he should have 
good knov/ledge from the fields of mathematics, numerical analysis 
and mathematical statistics and some experience v/ith problems of 
education and psychology rather than the opposite (be a behavioural 
scientist with informal experience in programming). This is because he 
must be able to develop new programs along with his other services to 
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the clients. I consider thic ac highly important: my experience with 
some available standard prograrrc iz not good and in othei inctanccs 
there are no such programs at hand. (I am consciouG of the fact that 
such expcrienccy can be very different fron? place to place due to 
different software facilities. ) Again, it takes time and cuts down the 
number of clients which the programmer can serve. 

The last three points of the adviser's list concern his own further 
education. The art of keeping oneself informed on literature is, as 
every researcher knows, not easy. Besides different abstracts, review 
articles of Annu. Rev. Psychol, and Rev. educ. Res. may constituc 
good starting-points, but continuous scanning of certain journals cannot 
be avoided. Of the statistical ones I have found Ann. math. Statist. , J. 
Anier. statist. Ass. and J. roy. statist. Soc. most useful. Other 
journals not belonging to behavioural sciences and which arc of more or 
less permanent interest arc Biometrics and Biometrika. Among our own 
journals you may find items of methodological interest here and there, 
but I have found the likelihood of this to be greatest in the following 
journals: Brit. J. mi.ui. Gcatist. Psychol., Educ. psychol. Measmt, 
J. math. Psychol,, Multiv. bchav. Res. and Psychometrika. I may also 
add Bchav. Sci. and Econometrica. 

Of course, it is also important to read books on research methodolo- 
gy, of which I have already mentioned some. However, there are, with 
some exceptions, not many new things to be found in textbooks. I have 
sometimes wondered how books on what is called measurements and 
elementury statistics can be released in such numbers as is the case 
today. They are evidently attractive, because I do not think that 
publishing companies are economic fools. Is there really an increasing 
number of people who read tliesc books or do the same people who have 
not managed to grasp the content and therefore, with an admiring 
optimism, hasten from book to book in the vain hope of eventually under- 
standing the subject? If the last presumption is correct, due to insufficient 
basic knowledge of the reader and/or pedagogical failures of the books, 
much has to be reconsidered. 

I do not intend to waste many words on point six: to take part in 
conferences and other valuable contacts. One can say a great deal 
concerning tlie construction of conferences, such ac the number of 
members and the limitation of the subject, but most researchers agree 
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that pcrconal contacts are very ir-portaiit. This ic particularly true for 
the isolated specialist, like some advisers on research ;^ie\:hodology, 
because his daily work seldora makes it possible for him to have 
mutually profitable converjjations within his own area of specialization 
with colleagues. 

The adviser's own researc 



The adviser can do researc 
As I see it, there are three 

1. The orn^anization of re 

2. The use of known methc 

3. The development or int 



h of his own on many different problems, 
main ar^as : 
search methodology, 
.ds, 

•eduction of new methods. 
The first area will perhaps surprise some people, but I believe it 
to be essential to examine how research methodology should be organized. 
The methodological quality is dependent on an efficient system to spread 
methodological knowledge. t/[any m.odels for this are conceivable, as are 
criteria for evaluating such models. For instance, one can have some 
special institutes doing methodological research or its m.embers can be 
allocated at different educational and psychological institutes, the degree 
of specialization of these members ha.s to be discussed and this goes for 
the teaching of methodology too. 

Criteria are not easily produced, but one simple way to get some is 
just to ask researchers about what they need and what they are missing. 
Besides the difficulty of defining a researcher, which must be done under 
all circumstances, there is the risk of getting meager results because 
some researchers do not see any problems here, for different reasons. 
Another way could be to do a content analysis of what has been written about 
these problems or let som.e experts do the job; botli attacks can also have 
several drawbacks. Anyhow, some criteria are more disputable than 
others. For instance, few researchers will deny that experiments are 
preferable to non-experiments, other things being equal, while the proper 
choice between parametric and non-parametric statistics will certainly 
divide researchers on many applications. So you may be able to write 
down some 'imequivocaP criteria, take a sample of reports (which by 
the way requires a definition of a population) and report your results. 
Something similar to this has tfeen done by Stanley (1967), whose paper 
also takes up some other points which I have touched upon. 
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The cccond point is most in line with the purpocc of the Cocicty of 
Multivariate Experimental Pcycholon;y: to develop r.->ethoctc through 
feedback from the experience of tlicir ui>o. I have already mentioned 
something: the problems of data analysis and the problems about 
similarities and differences of methods. Much ic done on measurement 
theory which is not used in field applications and I suspect that different 
results in a special area may very well be the effect of different res;?ar- 
chors realizing their concepts on different scales. V/hy not make a study 
of this to learn the consequences of different scales? Or to take another 
example: what would be the consequences of doing all the t-tests (or the 
ANOVAs) instead of the factorial discriminant analysis in the situation 
mentioned before? And still another one: what Cattell (1952) has once 
named R and Q analysis are in some sense the same analyses, but from 
otlier points of view one of them is better than the other. \7hy is tlie 
individual- space used most of tlie time? Can we use e.g. Q discriminant 
analysis for anything ? 

Perhaps somebody wants to include the answer to the last question 
within the development of nev/ metliods, and the boundaries are indeed 
very fluent. This reflects reality, for experience with known methods 
points to certain things, v/here more research work seems to pay off, 
tliat is to develop methods further. (This is valid for programs too. ) 
One principal dividing point, concerning statistical methods, is whether 
they involve latent classes, clusters, etc. or not. I/J.etliods which comprise 
manifest classes have been examined to a greater extent than the other 
kind of methods (the split-up methods), but there remains a great deal 
to be done. For example, methods like canonical correlation analysis 
are usually based on some optimalit^r criteria which may be quite 
insensitive: very different weights can still give a correlation in the 
neighbourhood of the optimal correlation. Many times one is interested 
in laiowing whether weights with special restrictions (e.g. the two weight 
vectors defining the first canonical correlation are equal) produce a 
correlation sufficiently near the optimal one. I believe that this kind of 
hypothesis testing is both useful and underdeveloped. I have seen a test 
for fully assigned weights of discriminemt analysis (Rao, 1965, pp. 482- 
483), but I do not know if this has been extended to arbitrary restrictions, 
e.g. v/itli tlie help of the generalized likelihood-ratio criterion. 

The split-up metlio.ds are often baised on such things as the axiom of 
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local independence, the mixture of multivariate normal cictributionc . 
discriminant analysis criteria or combinations of thocc. They h::ve the 
common property of producirx^ licmosencous classec which differ from 
each other. Other types of cplit-up methods aim at multivariate 
matching, e.g. individuals are to be matched in quartets on the basis 
of several variables or items arc to be clustered into three parallel 
tests. Of course, you may also imagine combinations of split-up methods 
and methods with manifest classes. V/hile I can see some meaning in 
performing e.g. a 2 x 2 factorial design on factor analysis I cannot see 
the significance of a latent profile analysis which produces a 2 x 2 
factorial design or any other latent design. Anyhow, the split-up methods 
offer a tremendous amount of problems. 

There are several other developments and/or introductions which 
can be indicated. Such an area concerns checks on agreem.ent between 
judges, and the area needs to be cleared up* It seems to me that the 
attacks on this problem are almost as numerous as the authors 
describing it* Multivariate non -parametric statistics is another area 
which we should use to a greater extent. As is clear from Aitkin (1971, 
p, 248) something is already done and 'the systematic development of 
these procedures should produce important shifts in emphasis in the 
use and teaching of statistics' . One may also wonder how long time it 
will take until we dare to use Bayesian statistics, which has produced 
a metatheoretical issue that has been debated with sometimes great 
heat. Much has been done in this area, see e,g» De Groot (1970), but 
we hardly use it. (Strange to say, v/e instead use several Bayesian 
ideas in applications as e*g* psychometry and behavioural decision 
theory, which is clear from Novick & Jackson, 1970, and Clovic & 
Lichtcnstein, 1971, respectively.) Are we scared by tlie orthodox 
statisticians' objections, or do we not expect ourselves to be capable of 
honestly assessing prior probabilities? 

FINAL DISCUSSION 

Simple problems mean simple statistical methods, complex methods 
are required for complex pro^blems and when the com.plexity of the 
problems grows the behavioural researcher has usually not the knowledge 
to match the problem with an appropriate method. It ic e,g, not unusual 
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to find rather complex clcsi;;,nc v/hich use too cimplc tcctG and as more 
and better programs becoir.e available you cannot blame those, (I think 
that the failure of matching complexity of problcrnc with methods goes 
for the construction of mathematical models as welL However, I have 
not defined research methodology to include constructions c£ theories, 
etc) 

In.competent use of measurement theory, psychomv-try and statistics 
cannot but lead bo bad research, the consequences of which are not 
easily predicted. The remedy fcr this is better mathematical training 
of the behavioural scientist and the creation of special, well-educated 
advisers on research methodology. This idea is by no means new, 
because it is pretty close to what Stanley (1967) tells about Laboratory 
of Experimental Design, Also, several aspects here are in agreement 
with those in Harnquist (1972), whose paper treats a more general 
subject. But it is quite a different thing to induce researchers to admit 
that these bad methodological conditions exist and still another to get 
funds for improving them, (According to my own experience, the latter 
seems to be the most difficult task, ) I am afraid that researchers in 
education and psychology are slowly di_y.iding themselves into t\^'o camps 
depending on knowledge of mathematics, statistics, etc, and that 
these two camps have increasing difficulties in communicating with 
each other. This process must be stopped and I do not intend to propose 
that to be stopped by neglecting all statistics, etc, 

l/Ly purpose with this paper has only been to draw the attention to 
some methodological problems within education and psychology, I have 
also given some hints here och there how some of the problems may be 
solved. However, unless more researchers are willing to change their 
attitudes to methodology, very little can be done. There are still too 
many researchers who look upon statistics, etc, with suspicioun and 
who shov/ resistance to learning and applying better methods. Some will 
acknowledge good knowledge of research methodology with their tongues 
but not with their hearts, .raying 'multivariate analysis is a fine thing 
but have no use for it' , Three possibilities come into my mind: either 
they will not admit their inadequate knowledge and/or they arc not able 
to see any problems here, or these problems are only figments of my 
imagination. If the third cause is the correct one, the content of this 
paper will corne in for severe criticism. It will, in such a case, be very 
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inter CG ting to follow the line of proof. If there are no reliable counter- 
arguments, it is about time to cct free resources to improve research 
methodology. 
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